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A computed f^naged instruction (cyi) package to teach 
BASIC programming, together Vith questions to be answered ia its 
evaluation, described. The training package consists of 16 ^ 
learning modules coupled; an interactive CHI system 'generates 
ei^etests and postt^s%s for each module and branches students to the 
next appropriate module. It is intended 'to be sold to customer^, to be 
used in a self-paced^ aode. Evaluat'ion questions pertain to (1) nhat 
percentage of custoniierd vill use the CHI system, (2) vhejther students 
will use the tests as^intended, (3) whether a sequential probability 
test ratio ,iriljl reduce testVlength significantly, (tty\ whether test , 
reliability is maintained, a\^d (5) whether all test i/ims are working 
ptoperly* {Author/CHV) 
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EVALUATING INTERACTIVE, COMPUTER-MANAGED INSTRUCTION 

Jesse M. Hemes 



.ABSTRACT ' * , 

This paper aescrirrto tne evaluation of a new training pacKage,>Dn 
6ASIC language programming. 'The traini^ng package consists, of 16 
learning moauxes coupj.ed with an interactive Computer-Managed^ 

/Ihstruction (CMI)^ system. The CMI .systenr generate^ pretests and' 
posttests . tor eact) of the moauXes in the course. The entire 
training package is intended to be sold to customers to be use^ 

- in ,a self-pacBd mode. Tnife paper aiscusses the strategies that 
are being employed to evaluate this approach to the quality 
control ot se if -paced instruction. 
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THE BASIC PRIMER: COMPONENT - PARTS 



The BASIC Primer is -a new seif-paced^ course on BASIC language 
programming . This course consists of 16 modules which cover 
topics trom simple input/output and computations to the! use of 
sequential and virtual array files* Each module has j dearly 
stated behavorial object^ives, text and graphics to .instruct the 
learner on the topics being presented, and exercises to rbe per- ' 
formed both on pap^r and on a computer system to reinforce t*he 
cobcept? being presented. • \ ^ 

Before each module, students are given the opportunity to take a 
pretest on that tnodule. ^ If they "pass" this pretest, they, are 
instructed tp go on to another module. If they do 'not pass the * 
pretest,," they are ask&fS^ Xo ^tudy the module and then take a 
posttest . Students whcr pass this posttest are routed to the^next 
appropriate module . Stud^r^ts who do pass the posttest are 

told the numbe^ of the ^Abjectives ■ on Wnich they missed^ i-teirs/ 
^ate asked to study thes;6 objectives again and do any exercises 
that they skipped, and Jtlaen take another posttest. This process , 
is tlowcharted in Figuc4 v.* \ . 
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All of the. (tests 
ys^em. 



for tbis course are stored on the student 's - own 
computer sy^em. This can be "a large timesharing system or a 
small stand-alone system. The software developed for ' this study 
is ^^itten in a subset of the BASIC ianguageV It can theretor^ 
be translated to a variety of operating systems quite easily. 



CMI SOFTWARE 



The computer^managed instruction system consists of three ^main 
programs. These are the Registrati^on Program^ (CMI), the Router 
Program (ROUTER) , and the Computer-Assisted Test Administration 
Program (CATSTR) . Two other , subprograms are also included in 
this system; the New Stu^^jent' Registration Program (REGSTR) and a 
Feedback Program (FEECBKf. Students move through -these programs 
as illustrated in Fi^re 2. - . ^ ^ 
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The Reqistrgtion Program ♦ , ' 

The m^*in Registration Program {CMI) 6^es tvo basic things. 
First, it asks the user tgr his or her code, name. Thi*s uniquely 
identifies .each student so that the dat^ stored on his ot her 
work will be contidential* Second, it 'uses this 'code name to 
search for student-^specif ic data in the rostet file* This data 
includes the student's tirst and last names, his or her course, 
and the type of terminal that that student is using. if a stu^ 
dent has not yet registeofed on the CMI system and selected g code 
name, he* or she is directed, to press the RETURN key Without 
typing anything else. ^<rhis causes tlie^ system to branch to the 
New^Student Registration Program (REGSTR) . 

The New Student- Registration Program allows students to register 
tjhemselves on the system. This feature is required because the 
^tjraining package is intended to be used without an instructor. 
The students enter their terminal types, their names, and their 
addresses. They are then asked several demographic questions 
regarding their age and education and motivajtion for taking the 
coarse* This 'd^emographic data will allow us to look^ for trends 
in student achievement and correlate these with certaih student 
factors. 




Theypurpose of the Hout^^r program (ROUTER) is ^to identify the 
test that the stydenjt-^ls about to take * ^This 'is not always a 
sjpple matter* The 3^6 modules in The BASIC Primer have specific 
^requisite relationships. These prerequisite relatipnships are 
iown in the module m^p* in Figure 3. j 

, i%. 

This module map is read in the tollowing manXei? * First, one 
begins at the bottom* After completing Module 1, one moves up 
the map to Module 2. But fetter completing Module 2, one has the 
choice of studying either Module *3 or Module 4 because 2 is the 
only prerequisite to each of these miodul&s. 'A ditterent kind of 
relationship exists between llocJules 12, '13,' and 16* \ln this- 
case, a student must comjiiete both Modules' 12 and 13 biet^te he or 
she can tackle Mod i^Jre 16. . ^ 

Thp Router Program has an algorithm which reads .in the jt'terequi-* 
site modules ror each of the 16 modules in ttie course and stores 
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theft in- an array. - This array, combined witlT the student status 
on e^ch of the modules in the course,^ is used to determine for 
which modules the student has met the prerequisites. , If the 
student has met the prerequisites for only one module, the appro- 
priate pre*test or* posttest for that module is ^generated. Ifv a 
student has met the prerequisites for -wire than on^ module, he or 
she is given the choice of, the module to tested* ^^-In this way, 

•the system assures Jthat studeats* go through .the course in the 
manner in which j.t was intended.' This constraint strengthens the 
instructional design of the course by guaranteeing ^ that each 
student possesses all of the prerequisite knowledge required for 
e^ch module that he or she is going .tjo* study. The system does 
not allow a student to take a test on a jnodule for which he or 

;she has not mastered ail of th^ prerequisite modules. 




Figure 3 
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The Router Program also gives ' students the option of displaying - 
tneir . statuses .on each of the modules in the course. Tnese 
st^uses are reported as; ' . 

• Not attempted ' ^ * ^ 

• Pretest tried but not completed satisfactorily' » 

• Pretest completed satistactorily ; posttest skipped 
Posttest tried but nq.t completed satisfactorily 

• ^ Posttest completed satistactorily 

Students are allowed to take a' pretest' for a i^pdule only once. 
All sulbsequent tests are aUtomaticallS^ interpreted as post tests , 
because it is assumed that if a student does not pass a pretest 
he or she will go and study the corresponding moduJle. Students 
are allowed to take posttests as many times as necessary to^ 
complete them satisjatitorily , i*e., demonstrate mastery on all>of' 
the objectives in that , module . (Tests are generated interact 
tively in real time, so no two tests are e^^actly alike. See the 
dj^scussion* below . ) , * 



The Testing P rogra m * 

[ / . • ' . 

The Computer-Assisted Test ^Administration ^Program (CA^fSTR) gener- 
ates botp prje tests and posttests. Tnese bests are administered 
to students at a computer 'terminal. The purpose of the Computer-, 
Assisted Test Administration Program is ti>-classif y the student , 
ats eitjher a masber or a non^master on the specific module be'ing 
teisted f and to make tnis classit ication in a minimum amount of 
time. To accomplUsh this, the Test ^Administration Program evalu- 
ates each stji'dent * response with a sequential probability test 
ratio . This algor ithm is discussed briefly below, but has been 
described elsewnere in detail (Hemes, iy78a and l^J78b). 

The algorithm evaluates a student ' s score after each item is 
presented.' 'Bteiis evaluation is used to cfl-assify the student in . 
one of three categories. It the -student's score .Exceeds a map- 
tery tnteshold with a predefined level* of contidencev the student 
is classified as a master dnd testing is terminated. . If the 
student's score is less 'than a non-mastery .injfeshold with - a 
specified degree of confidence, the*" student fts classified as- a 
non-master. If the student 's score tails witKin tnese two 
thresholds, another item is presented. Once the student rispd^ds 
to this item, the evaluation algorithm is repeated*. . \ 

Using the sequential probability 'test ratio, it is possible for 
student to remain forever in tne "uncertairfty band" between th^ 
mastery and non-mastery thresholds. ' For\this Reason,- testing is" / 
terminated atter 30 items have been/presei)ted regardless ot tht 
^studept' s score . At this point, the student is^. classified as 
either a master or a non-mastet by determining the thresnajd tnat" 
his or her current score is closest to. * * \ 
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Thirty is a rather large number of items for each module . To 
make the tests truly repeatable, more than 30 items are needed in 
the item bank* for each m'odule so that the randomised selection 
process does not prpduce identical tests'. Fo"?^ The BASIC grimer ^ 
the^^are 754 items in the entiie bank. This ^allb^ an average 
of .about 45 items tor each module. Since^ it is expected that 
very few te^ts will actually bfe 30u^itemp ^long , this av&rage is 
suffici^t to yield a very large^ number of datterent test torms. 

Students who demonstrate mastery on a test are branche^S back to 
the Router Program. - Th^y then select the next* module on which 
they would like to be tested. Students who do not demonstrate 
mastery on , a test are told the ob3ect;ives on Which they missed 
items t The CMI system then halts, and students ' are directed ti> 
do additional study otf-line. 

Students who haye completed all of the modules ii> the course are 
^branched to the Feedback Program- This program asks for atudent 
comments on the course and generates another data listing f ile . 
This* li&ting tile verities that*the student has earned a diploma 
and contains othei: data that is needeiB by the course development 
group to evaluate the training package. Students are directed to 
mail tjiis file ba?k to the course development group either on a 
machrne-re^dable media, or they are asked to print this tile on 
p^per and mail it. In either case, this data is' combined^ with, 
data from other students in o^ther training locations by a large 
system within the course development group. It is tfhis feedback 
data that is us^d to €valuate;^the training package as a whole. 



' EVALUATION QUESTIONS AND STRATEGIES 

The data described in the pr^geding section will be used to try 
to answer five research questions oh the system. EsTch of these' 
questions and the strategy that will be used to evaluate the data 
ife described in the paragraphs that follow. ^ . - 

(1) Wtiat percental of ^ customers who study- the training materi- 
*als Will use the CMI system? 

i . • * 

^ There are two ways^ in which we will know how many students 
'are using The BASIC Primer . . First, w^ will have data on 'the 
^ number of CftI systems' in use from our corporate sales data. 
Second, we will have some id'ea of the number Qf students 
taking the course by the registration forms that are gener- 
ated by the New Registration Program. These two pieces -of 
data, taken together, shoul<J give us a good idea of the num- 
ber of people ^ho at lea^t began *using the system. 

After any studeK^ has completed the first ten modules of tlie 
, . course,^ a new listing tile will be generated for 6hat^ stu- 
demt. This fii^ will contain the status data on all of the 
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students who are registered on that system. Frqni this data* 
* we wi^Ll be able to ascertain ftow ; ipany students began the 
course but ^id ' not complete *it. That is> we should be '/able 
^to know how many students began using *the testing system and 
did * not \continue. if we find that a lar'ge number . of stuj 
^dents' tall into this category, it wall be nec^ssa^ry tor us 
to iavestigate further the reasons that they ^disco^ntinued ^ 
the ^use of the CMI system. Most importantly, we' would like 
to know whether they, continued stjudyiog the module? but just 
gave up studying the tests. If ''this is th^ case/ it will be, 
clear that the CMI system did not fulfill its goal. On the 
other hand, if we tind ,that 'rfost students^did continue to 
use the Cm system, the status data generat;ed at this ^int 
, will give ^us an excellent picture of how the system and 
mastery algorithm are working. 

^ (2) 'Will those students who use the CMI system use the t^sts as 



intended? 



r 



ERIC 



^n addition to updating the. status records foi; a particular^, 
student each time that^ he or^ahe takes the testf the system 
will maintain s 'test history file. This file, will record 
• - data^ on each test that is administered in chronological 
order. ^E^ch test history record will- record 

• the .student number, . - 

• the module being tested, ^ 

• whether - the test being, administered is a 
, ^ ^ pretest or a posttest/ , ^ ^ 

• whether the test beiing ^administered is a 
^ *normal test or, a special 30 itefe test (used 

, to^ reliaMJ^ty measurements,; See ,belote) , 
and 

4 the date and time. that the test ^as started. ' 

. ' Thus data will allow us to asce^>ain whether a student as^ 
just retaking tests one right after the ,other.or actually 
^ studying between ' test administration^. .V*e will assume that 
, if B student repeats a test in le^s than, ten minutes,^ that 
he or she did not res^iwrn ta the materials betore re(Jue sting ^ 
iSf a rete*st. On the other hand, if we find that the tirife 
between^ repeated tests is on the order of 15 to 30 minutes, 
we will assume that the student .did* look over the material 
^ and possibly do addl^txonal exercises betore he. t>r she re- 
' turtied to the system for a petest. 

" The test history tile will also' tell us the order in which, 
s±-^ents go through the course modules. It will provide a 
-trali^ which we can analyze to assure that the Router *Progr,am ; 
did. not allow students to go on to wore advanced modules 
betore they met the prerequisites. 
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(3) Does the^ sequential probability . test ratio ^ reduce test 
lengths significantly? , 

4 Each module on tfee system has an associated log file^ Qne 
* of the types of data that is store^ in this log tile is" the 
length of each of the tests that is talcen. Thesei lengths 
are cecorded in tour ' categories. They *are recorded for , 
^ masteirs on protests, non-masters on pretests, masters on 
^ * , posttests, aod non^masters on posttests'.^ If the sequential 
probability test raticT works' prope-rly, we should find that, 
the lengths of non^master tests'' are signiticahtly less thah 
the lengths of mast,^^ .tests. In addition, we should find 
^ that pj;etests on which ma;Ster'y decision^ were made are, on 
^ the average, longer than ' posttests on which mastery deci- 
sions are made. This, result should occur because the cri- 
teria for mastery on^ pretests are more stringent than ^he 
mastery criteria on pgsttests. In, addition, we hope to tind 
that the* average l^engtK of a test is considerably shorter 
than S0 items. Tnis will indicate . that the setjuential 
^probability test ratio- was gsef'ul in reducing test lengtljs. 

. ' Two' additional items that are stored Jor/feach student wilU 
. ^ hejp us ascerterin whether any reductfpn in test length dOe 
to' the sequential ^probability test ratio is signiticant. 
Thes^e items are the total amount of time that eac*h student 
spent ^testrng, afrd the to€al number of teat items t^ft w^re 
^ presented tdf- that student. By divid ing the total time by 
the total number of items, we will have an i*ftdicator of the 
average amjDunt of t^'me that it takes for a Student to re- 
sppnd *i to a single test i tem^ By looking at the^numper of 
.t^sts that the -student has taken and the aiftount of' time that^ 
would hav^ been required if Jthese i had befen tixed length* 
fees.ts of^s^. 20 or' 30 items, we will- have an indicator of 
the amount of time ssTved by using the seijuential probability 
test ratio. Again, the decision algorithm will prove itself 
useful* if the amount of time . saved tis a signi^ticant portion 
of the total ^ainount of ■ time sp3?nt testing. ^ 

(4) Is test reliability maintained ^PJX when the sequential 
probability ^^st ratio* causes shoTltlbests to be adminis^ 

To oT:itaih a measure of^the tests'" reriab(tf.ity , every fifth 
tes^ that is administered* t;'d a student will be. forced tO'30 
items long regirtrdle&s * of * the. results- of, the sequential 
probability te^st ratio. ' When this occur^, the System' will 
record two 'decisions for the student. Jt'will first ^record 
the decision that it would have fhade if ,-testing h^ b^en 
terminated as soon as po?siJ:>le by applying t^ie' ^seq^itentiail 
firobability test ratio. It will then continue ^tfie test and 
record the decision that is made after 30 items have^ been 
presented. Thei^e two decisions will be *compar^d in a *2 x 2 
contingency table. tfhe analysis of t})is contingency table 
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will be made by applying the G index . This ind^ex is a 
refinement by Livingston (1976) of the percentage -of ^agree- 
ment first described by Carver (1970). 
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Using the contingency fable in Figure A, Carver defined the 
percentage of agreement as: ' . ■ ; ■' 



P0 = 



a+d 



a+b+Q+d ** . ' ■ ' . 

This measurement varies betw^n 0 and 1. ■ ■ 

Swaminathan et al. (1974) agreed with t:arver*fe c6ncept <o£ 
'^criterion-referenced reliability as a coeffi'cient of classi- 
fication", ^but prCTer r^ a mone sopmsticated Computation^ 
known as the kappa coefficient. Swe&ey ^d Pear Istein 
(1975) preterred an even^ morfe sopnisticated approach" known 
as^ the * phi coefficient. Livingston's coefficient, . the G 
index, is simply: * * / ' ' 



G ^ 2 X {P0 - 0. 5) " * 

^This confident va"?i^s between -1 and +1, apd 
compatit^e with thQ> -kappa and phi coetticients 
tailed discussion of thts^mathematics is provi 
/i974b.) ' . ^ ^ 

4 . . ■ / 



is therefore 
(A mbre de- 
d^d in Heines,. 



Digital Educational Services \" . page 10 

Technical Report No* 2 \ , ^ 



5t iki 



(5) Are all, of the test item? on the systemworking properly? 



-^n additiqn to' the test length data, thg log tile for each 
modtile also records item ahalysis data* This -data* is "^a 
tally of the n'ftmber of times of that each* i^esponse was 
chosen for each question presented* Liice the fest length 
data/ the item analysis, data is separated into tout 'Cata- 
g oriels* That ^is, tallies) a^e kept' separately^ for 

• pretests o^ wh]*t:h mastei;y decisions are made, 

• pretest^' on which non-mastery decisigps were ^ 

made, ' ' ; ■ > " 

• posttests on which mastery decisions wer>^made/ ^ 

and ' . * ' ; ■ " • 

posttests on whicH .non-mastery decisiorTS\ were 
V ^ made* . y f 

*'This, allows to compute prete^/postfcest ^^ n d^ ■master/non7 
master discrimination^j^ndaces* Theretore^ we will toe ,ablje 
to ascertain how welT each item discriminates ^between mas- 
ters ^and non-mas.ters as well as how well it discriminates 
between pretests and posttests* The log .data from vaViqbs 
tI^alnlng^ sx'tes will be^ combined t© Vive us a Marge .data 
sample * This ^should make the item analysis very signifi- 
cant* ■ ^' ^ 



GLOBAL ^ISSUES ' - * \ 

^ The BASIC Ptimer is bher|taj:st compute r-pased course being ottered 
by Digital Equipment Corporation^ we feel thai the use of tihe 
computer in- our current mode gives us tl^e^power ' tjj evaluate gur 
training materials objectively* Nev^r- before have been able^ 
to collect this- type of data ^on a- training course^* ^If* the CHl' 
system fulfills its goals, it will ^not ORly* improve the quality 
control of our courses,, but ••it will ^provide ts with a vehicle " for 
improving all of the course^s that we o£f er in this mode* ThC 
BASIC Primer and it^, complementary CMI system ar^ currently being 
tested^ in selected Jites in New England* The fcinal y^ersion of 
this tlraj^ning pacKage should" be availa^e to ccTsrjtiomers ih the 
3pring of ,1379*^ - ' . - ^ ■ > 
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